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Why GAO Did This Study 

The U.S. Customs Service is 
conducting a multiyear, multibillion 
dollar project, the Automated 
Commercial Environment (ACE), a 
new trade processing system that is 
planned to support effective and 
efficient movement of goods into 
the United States. By congressional 
mandate, Customs' expenditure 
plans for ACE must meet certain 
conditions, including being 
reviewed by GAO. This study 
addresses whether Customs' latest 
plan satisfies these conditions and 
provides observations about the 
plan and Customs' efforts to 
implement GAO's open 
recommendations for improving 
ACE management. 



What GAO Recommends 



To ensure that Customs is 
positioned to effectively manage 
the acquisition of ACE, GAO is 
making recommendations to the 
commissioner aimed at improving 
Customs' acquisition management 
capabilities. Customs concurred 
with GAO's recommendations and 
described specific actions that it is 
taking to respond to each. 



What GAO Found 

Customs' November 2002 ACE expenditure plan, the fourth in a series of 
legislatively required plans, provides for certain project management tasks 
as well as the definition, design, and development of the first release of the 
second of four planned ACE increments. GAO's analysis of the plan shows 
that it meets the legislative conditions imposed by the Congress. 

In its series of reports on Customs' management of ACE, GAO has made a 
number of recommendations, which Customs is currently addressing. 
However, Customs has been slow to correct weaknesses in two areas 
fundamental to effective acquisition management — people and processes. 
These weaknesses increase the risk that ACE will be late, cost more than 
necessary, and not perform as intended. 

• While the Customs Modernization Office (CMO) has developed a human 
capital strategy, neither the strategy nor supporting documentation 
identifies how the commitments made in the strategy will be met, 
including what steps will be taken and what resources are needed to 
execute the steps. Further, based on the strategy's timeline, it will be 
over a year before the CMO fully implements the strategy. 

• Customs' has made slow progress in implementing key acquisition 
practices, such as project office management and acquisition risk 
management, which GAO first recommended Customs do in 1999. Since 
that time many practices have been developed, but almost none of them 
have been implemented, as shown in the figure below. 

Additionally, Customs' ACE contractor is conducting system tests in 
accordance with best practices, initial test results are positive, and 
indicators suggest the contractor is delivering a quality product. However, 
Customs is not employing independent verification and validation (TV&V) in 
overseeing ACE testing, which is one way to mitigate the acquisition 
weaknesses cited above. Without IV&V, Customs states that it is relying on 
the contractor's reputation and maturity level as guarantors of system 
quality, which is not adequate for a complex and risky program like ACE. 



www.gao.gov/cgi-bin/getrpt?GAO-03-406. 

To view the full report, including the scope 
and methodology, click on the link above. 
For more information, contact Randolph C. 
Hite at (202) 512-3439 or hiter@gao.gov. 
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United States General Accounting Office 
Washington, D.C. 20548 



February 28, 2003 

The Honorable Ben Nighthorse Campbell 
Chairman 

The Honorable Byron L. Dorgan 
Ranking Minority Member 

Subcommittee on Treasury and General Government 
Committee on Appropriations 
United States Senate 

The Honorable Ernest J. Istook, Jr. 
Chairman 

The Honorable Steny H. Hoyer 
Ranking Minority Member 

Subcommittee on Transportation, Treasury, Postal Service, 

and General Government 
Committee on Appropriations 
House of Representatives 

In November 2002, the U.S. Customs Service submitted to Congress its 
fourth expenditure plan, seeking release of $314 million for its Automated 
Commercial Environment (ACE) project. ACE is to be Customs' new trade 
processing system and the first project under the Customs Modernization 
Program. As required by Customs' fiscal year 2002 appropriation and 
continuing resolutions, 1 we reviewed the expenditure plan. Our objectives 
were (1) to determine whether the fourth ACE expenditure plan satisfies 
certain legislative conditions, (2) to determine whether the plan is 
consistent with our open ACE recommendations, and (3) to provide 
observations about the plan and Customs' management of ACE. 

On December 27, 2002, we briefed your offices on the results of this review. 
This report transmits the results of our work. The full briefing, including 
our scope and methodology, is reprinted as appendix I. 



x For example, P. L. 108-2 appropriated such amounts as may be necessary under the 
authority and conditions provided in the applicable appropriations act for fiscal year 2002 
for continuing projects or activities that were conducted in fiscal year 2002, at a rate for 
operations not exceeding the current rate, and for which appropriations, funds, or other 
authority was made available in the Treasury and General Government Appropriations Act, 
2002, P. L. 107-67, 115 Stat. 514, 520 (2001). These appropriations shall be available to the 
extent and in the manner which would be provided by the fiscal year 2002 Treasury 
Appropriations Act. 



Page 1 



GAO-03-406 Customs Service Modernization 



Concerning our first objective, Customs' expenditure plan satisfies the 
legislative conditions specified in Customs' appropriations act. That is, the 
plan provides for (1) meeting the capital planning and investment control 
review requirements of the Office of Management and Budget (OMB); 
(2) complying with Customs' enterprise architecture; 2 and (3) complying 
with federal acquisition rules, requirements, guidelines, and systems 
acquisition management practices. Further, the plan was reviewed and 
approved by the Joint Capital Investment Review Board 3 and OMB. 

Concerning our second objective, Customs is making progress in 
addressing our open recommendations, which are as follows: 

• Justify and make investment decisions incrementally. 

• Before building each ACE release, certify to Customs' appropriations 
subcommittees that Customs' enterprise architecture has been 
sufficiently extended and updated. 

• Develop and implement a rigorous and analytically verifiable cost 
estimating program that embodies the tenets of effective estimating as 
defined in the Software Engineering Institute's (SEI) institutional and 
project-specific estimating guidance. 

• Immediately develop and implement a human capital management 
strategy for the Customs Modernization Office (CMO). 

• Limit future expenditure plan requests for management reserve funds to 
10 percent of the total funds requested for the program, or adequately 
justify any management reserve requests in excess of 10 percent. 



2 An enterprise architecture is an institutional blueprint for guiding and constraining 
investments in business process change and systems. 

3 To expedite reviews of ACE expenditure plans, the Customs and Treasury Investment 
Review Boards were consolidated to form the Joint Capital Investment Review Board. 
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• Develop and implement process controls consistent with SEI's Software 
Acquisition Capability Maturity Model (SA-CMM 4 ), and by September 30, 
2002, assess and report to Customs' appropriation subcommittees on 
the maturity of CMO's software acquisition process. 

• Address the risks associated with the accelerated ACE acquisition 
strategy and report on the strategy going forward and plans for 
mitigating the risks associated with this strategy 

While Customs has taken actions to address each of these 
recommendations, its actions to develop and implement a human capital 
strategy and acquisition management process controls were only partially 
consistent with our recommendations. Specifically, while the CMO 
developed a human capital strategy that specified high-level commitments 
and satisfied some best practices, there are some notable omissions, 
including how these commitments will be met. Further, based on the 
timeline in the strategy, it will be over a year before key practices are 
implemented. 

Also, while Customs reported to its House and Senate appropriations 
subcommittees on its development of SA-CMM processes, it did not report 
on implementation. Further, while many of the acquisition processes have 
been developed, Customs has not yet implemented most of them. 

Finally, we made the following observations: 

• System integration and acceptance testing performed by the ACE 
contractor is being conducted in accordance with best practices. The 
contractor has established a test organization and built an infrastructure 
for test activities. It has completed system integration and system 
acceptance tests (SAT) for ACE increment 1, release 1.1, and these test 
activities were consistent with test management best practices. 

• The results of SAT performed by the ACE contractor are positive. All but 
seven ACE increment 1, release 1.1 requirements successfully passed 



4 Capability Maturity Model (CMM) is a service mark of Carnegie Mellon University, and 
CMM is registered in the U.S. Patent and Trademark Office. The SA-CMM identifies key 
process areas that are necessary to effectively manage software-intensive system 
acquisitions. Level 2 is the second level of the SA-CMM's five-level scale; achieving this level 
means that an organization has the software acquisition rigor and discipline to repeat 
project successes. 
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their test cases for SAT. For those that did not pass, only noncritical 
system defects resulted, and acceptable workarounds were identified. 

• Data provided by the ACE contractor on all unresolved system defects 
for ACE increment 1, release 1.1, show a recent downward trend, 
suggesting that the contractor is delivering a quality product. 

• Customs' oversight of contractor testing does not provide for 
independent verification and validation (IV&V). Industry best practices 
include IV&V as a part of effective managerial oversight and control. 

• The potential for ACE infrastructure to support other Department of 
Homeland Security applications could affect both cost and schedule for 
ACE. Customs is one of several agencies to be merged into the 
Department of Homeland Security, and it has proposed that the ACE 
infrastructure be used by these agencies. Because a meaningful 
understanding of the effect of the new department's use of the ACE 
infrastructure is not yet known, there is the potential for existing ACE 
cost and schedule commitments to change. 



Recommendations for ^° ensure that Customs has the requisite capability to manage its ACE 
-r, , . A , . acquisition, we recommend that the Customs Service Commissioner 

uac^uuvc ri ^ llulL designate strengthening CMO human capital and acquisition processes as 

priority matters. To this end, we recommend that the Commissioner direct 
the Chief Information Officer (CIO) to immediately 



• develop and implement each of the missing SEI SA-CMM practices for 
the key process areas discussed in this report, and until this is 
accomplished, report to its appropriations subcommittees quarterly on 
the progress of its efforts to do so; 



• develop and implement the missing human capital management 
practices discussed in this report, and until this is accomplished, report 
to its appropriations committees quarterly on the progress of its efforts 
to do so; and 

• establish an IV&V function to assist Customs in overseeing contractor 
efforts, such as testing. 

Additionally, we recommend that the Commissioner take steps, as 
appropriate in light of Customs' merger into the Department of Homeland 
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Security, to have future ACE expenditure plans specifically address any 
proposals or plans, whether tentative or approved, for extending and using 
ACE infrastructure to support other homeland security applications, 
including any impact on ACE of such proposals and plans. 



In written comments on a draft of this report signed by the acting director, 
Office of Planning, Customs stated that it was pleased with the report, 
agreed with our recommendations, and described specific actions that are 
being taken to implement each. Customs' comments are reprinted in 
appendix II. 

Customs also provided additional information regarding our 
recommendation that it establish an IV&V function to assist in overseeing 
contractor efforts, such as testing. Specifically, Customs stated that it 
wanted to modify its earlier statement that it relies on the contractor's 
reputation and maturity level as guarantors of system quality. Customs said 
that its modified position is based on a number of practices that it believes 
are elements of an integrated approach to IV&V. However, the practices 
that Customs described as elements of IV&V, while additive to system 
quality, are not independent from program cost and schedule pressures, 
and therefore do not constitute IV&V 



We are sending copies of this report to the Chairmen and Ranking Minority 
Members of other Senate and House committees and subcommittees that 
have authorization and oversight responsibilities for the Customs Service. 
We are also sending copies to the Secretary of Homeland Security, the 
Secretary of the Treasury, the Commissioner of the Customs Service, and 
the Director of OMB. We also will make copies available to others upon 
request. In addition, the report will be available at no charge on the GAO 
Web site at http://www.gao.gov. 



Agency Comments and 
Our Evaluation 
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Should you or your staff have any questions on matters discussed in this 
report, please contact me at (202) 512-3439. 1 can also be reached by E-mail 
at HiteR@gao.gov. Key contributors to this report were Mark Bird, Harold 
Brumm, Jr., Barbara Collier, Scott Farrow, Joanne Fiorino, Tamra 
Goldstein, Michael Holland, Freda Paintsil, Madhav Panwar, Karen Richey, 
Randolph Tekeley, and Aaron Thorne. 




Randolph C. Hite 

Director, Information Technology Architecture 
and Systems Issues 
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Customs' Fourth Automated Commercial 
Environment (ACE) Expenditure Plan 



Customs' Fourth Automated Commercial 
Environment (ACE) Expenditure Plan 



Briefing for the staffs of the 
Subcommittee on Treasury and General Government 
Senate Committee on Appropriations 
and 

Subcommittee on Treasury, Postal Service, and General Government 
House Committee on Appropriations 

December 27, 2002 
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Customs' Fourth Automated Commercial 
Environment (ACE) Expenditure Plan 
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Results 
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Recommendations 
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Agency Comments 
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Introduction 



The U.S. Customs Service is acquiring a new trade processing system known as the Automated Commercial 
Environment (ACE). ACE is intended to 

• promote more efficient movement of legitimate trade and more effective enforcement of trade laws, 

• strengthen border security operations, and 

• include a single system interface between the trade community and the federal government, known as the 
International Trade Data System (ITDS), which is to reduce the data reporting burden placed on the trade 
community and facilitate the collection and dissemination of trade data for agencies with trade-related missions. 



ACE 



ITDS 



Functions unique 
to Customs 
operations 



Functions common 
to Customs and 
other federal agencies 



Functions unique to 
a single other 
federal agency 



Functions common to 
other federal agencies 



Common interface or "front end" (e.g., Web portal) for Customs, other federal agencies, and trade organizations 

— r 



T 



T 



i — r 



T 



Brokers and 
trade advisors 


Importers and 
exporters 


Carriers 





Agency 1 



Agency 2 



Agency 
104 



Trade community 
Source: GAO. 



Other federal agencies 
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Customs is required in its appropriations 1 not to obligate funds for ACE until it submits to the Committees on 
Appropriations ACE expenditure plans that 

• meet the capital planning and investment control review requirements of the Office of Management and Budget 
(OMB), including security planning; 

• comply with the Customs enterprise architecture; 

• comply with federal acquisition rules, requirements, guidelines, and systems acquisition management practices; 

• are reviewed and approved by the Customs Investment Review Board, Treasury, and OMB; and 

• are reviewed by GAO. 

Since 1999, we have conducted four reviews of Customs' ACE project, including our reviews of the first three 
expenditure plans. 2 In particular, in May 2002, we reported that ACE was a high-risk project and made 
recommendations for strengthening Customs' ACE management. Customs has implemented some of our 
recommendations and continues to work on others. 



1 Public Law 107-294 is the current continuing resolution providing funding for ACE through January 11, 2003. The continuing resolution appropriates such amounts 
as may be necessary under the authority and conditions provided in the applicable appropriations act for fiscal year 2002 for continuing projects or activities that 
were conducted in fiscal year 2002, at a rate for operations not exceeding the current rate, and for which appropriations, funds, or other authority was made 
available in the Treasury and General Government Appropriations Act, 2002. The continuing resolution further states that these appropriations shall be available "to 
the extent and in the manner which would be provided by" the FY 2002 Treasury Appropriations Act. 

2 U.S. General Accounting Office, Customs Service Modernization: Third Expenditure Plan Meets Legislative Conditions, but Cost Estimating Improvements 
Needed, GAO-02-908 (Washington, D.C.: Aug. 9, 2002); Customs Service Modernization: Management Improvements Needed on High-Risk Automated Commercial 
Environment Project, GAO-02-545 (Washington, D.C.: May 13, 2002); Customs Service Modernization: Results of Review of First Automated Commercial 
Environment Expenditure Plan, GAO-0 1-696 (Washington, D.C.: June 5, 2001); Customs Service Modernization: Serious Management and Technical Weaknesses 
Must Be Corrected, GAO/AIMD-99-41 (Washington, D.C.: Feb. 26, 1999). 

4 
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Customs' Fourth Automated Commercial 
Environment (ACE) Expenditure Plan 
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Objectives 

As agreed, our objectives were to 

• determine whether the fourth ACE expenditure plan satisfies the legislative conditions, 

• determine whether the plan is consistent with our open ACE recommendations, and 

• provide observations about the plan and Customs' management of ACE. 

Scope and Methodology 

To accomplish our objectives, we analyzed the plan and supporting documents and activities, comparing them to 
relevant federal requirements and guidance and applicable best practices. In particular, we compared the 

• ACE Increment 2 Cost Benefit Analysis to the Office of Management and Budget's (OMB) policy for cost benefit 
analysis of federal programs; 

• Customs Modernization Office (CMO) Human Capital Management Strategy to key principles from private 
sector organizations; 3 and 

• CMO's acquisition management improvement program to the Software Engineering Institute's (SEI) Software 
Acquisition Capability Maturity Model (SA-CMM); 4 and 

• test plans, test results, testing activities, and independent verification and validation (TV&v) activities to 
industry best practices as discussed in GAO's testing guide. 5 



3 U.S. General Accounting Office, Human Capital: Key Principles from Nine Private Sector Organizations, GAO/GGD-00-28 (Washington, D.C.: Jan. 31, 2000). 
1 Capability Maturity Model™ is a service mark of Carnegie Mellon University, and CMM is registered in the U.S. Patent and Trademark Office. The SA-CMM 
identifies key process areas that are necessary to effectively manage software-intensive system acquisitions. Level 2 is the second level of the SA-CMM's five-level 
scale; achieving this level means that an organization has the software acquisition rigor and discipline to repeat project successes. 
'U.S. General Accounting Office, Year 2000 Computing Crisis: A Testing Guide GAO/AIMD-10.1.21 (Washington, D.C.: November 1998). 

5 
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Scope and Methodology (cont.) 

We also interviewed ACE program management, support contractors, and modernization contractor officials to 
understand the scope and content of plans, clarify information in supporting documentation, and seek explanations 
for deviations from federal or commercial criteria. 

As agreed, we did not independently validate the project content and status information that Customs provided. 
However, we used Customs' provided data and the CostXpert software cost estimating model to check the 
reasonableness of the software development cost estimate contained in the expenditure plan. 

We performed our work from November 2002 through December 2002 in accordance with generally accepted 
government auditing standards. 



6 
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Customs' expenditure plan satisfies the legislative conditions. 



Legislative conditions 


Satisfies 




1. Meets OMB capital planning and investment control review requirements, 
including OMB Circular A-ll, part 3 6 






2. Complies with the Customs enterprise architecture 






3. Complies with federal acquisition rules, requirements, guidelines, and 
systems acquisition management practices 






4. Is reviewed and approved by the Customs Investment Review Board, 
Treasury, and OMB 






5. Is reviewed by GAO 







6 Customs provided us with a draft of the expenditure plan before it was submitted to the appropriations subcommittees. We determined that the draft plan did not 
satisfy the requirements of OMB Circular A-ll, part 3 (July 2001). After we discussed our analysis with Customs and support contractor officials, Customs amended 
the expenditure plan to satisfy the requirements before submitting it to the appropriations subcommittees. 

7 
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Customs' expenditure plan is consistent with our open recommendations. 



Open recommendations 


Consistent 


Status 




Fully j Partially 


Complete 1 In progress 


1. Justify and make investment decisions incrementally. 






2. Before building each ACE release, certify to Customs' 
appropriations subcommittees that Customs' enterprise 
architecture has been sufficiently extended and updated. 






3. Develop and implement a rigorous and analytically 
verifiable cost estimating program that embodies the 
tenets of effective estimating as defined in the Software 
Engineering Institute's (SEI) institutional and project- 
specific estimating guidance. 






4. Immediately develop and implement a Customs 

Modernization Office (CMO) human capital management 
strategy. 
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Open recommendations 


Consistent 


Status 




Fully j Partially 


Complete 1 In progress 


5. Limit future expenditure plan requests for management 
reserve funds to 10 percent of the total funds requested 
for the program, or adequately justify any management 
reserve requests in excess of 10 percent. 






6. Develop and implement process controls for SEI's SA- 
CMM, and by September 30, 2002, assess and report to 
Customs' appropriation subcommittees on CMO's 
software acquisition process maturity. 




S 


7. Address the risks associated with the accelerated ACE 
acquisition strategy and report on the strategy going 
forward and plans for mitigating the risks associated 
with this strategy. 


V 
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Other observations: 



Contractor-performed system integration and acceptance testing is conducted in accordance with best practices. 
Contractor-performed system acceptance test results are positive. 
System defect trends suggest contractor is delivering a quality product. 

Customs' oversight of contractor testing does not provide for independent verification and validation (IV&V). 

Potential for ACE infrastructure to support other Department of Homeland Security applications could effect ACE 
cost and schedule. 



Summary of recommendations: 



To ensure that Customs is positioned to effectively perform its role as the acquirer of ACE, we are making 
recommendations to the Customs Commissioner to expeditiously implement overdue people and process capability 
improvements and to keep its appropriations committees informed about ACE acquisition matters. 



10 
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ACE Acquisition Strategy 

On April 27, 2001, Customs awarded a 5-year, indefinite-delivery, indefinite-quantity contract to International Business 
Machines (IBM) Global Services. IBM and its subcontractors are collectively called the e-Customs Partnership (eCP). 
Through a series of contract task orders, Customs plans to acquire and implement ACE in four increments. The first 
two increments each have two releases. 

The following table summarizes primary ACE functions by increment and release. 



Functions 






ACE increment 7 








1 


2 




3 


4 




Release 1 


Release 2 


Release 1 


Release 2 




r 


Enterprise portal and IT infrastructure 


Initial 


Additional 


Additional 


Additional 


Additional 


Remaining 


Data processing (e.g., reference library and data query capabilities) 


Initial 


Additional 


Additional 


Additional 


Additional 


Remaining 


Trade agreement (e.g., manage visas, quotas; implement trade 
agreements) 










Initial 


Remaining 


Integration of ITDS functionality 




Initial 


Additional 


Additional 


Additional 


Remaining 


Account management (e.g., create, access accounts) 


Initial 


Additional 


Additional 


Additional 


Remaining 




Account revenue (e.g., generate billing statements, accept 
payments) 




Initial 


Additional 


Additional 


Additional 


Remaining 


Enforcement (e.g., determine violation severity, resolve compliance 
violations) 




Initial 


Additional 


Additional 


Remaining 




Selectivity criteria for assessing noncompliance risk, screening 
incoming transactions 






Initial 


Additional 


Additional 


Remaining 


Cargo release and tracking 




Initial 


Additional 


Additional 


Additional 


Remaining 


Ability to process multi-modal goods, full export functions, 
decommissioning of Automated Commercial System (ACS) 8 












All 



Source: GAO. 



' AU means all capabilities within this function are provided in this increment or release; initial means a beginning set of capabilities is provided, and the number 
and type of users may be limited for certain capabilities; additional means an increased set of capabilities is provided, and the number and type of users may be 
expanded for specific capabilities; remaining means that the final set of capabilities is provided to intended users. 

8 This system tracks, controls, and processes all goods imported into the United States. ACE is expected to provide all functions currently provided by this system. 

11 
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The following tables summarize complete and ongoing ACE contract task orders. The first table describes tasks 
associated with managing the Customs modernization program and the ACE project. The second describes tasks 
associated with developing and implementing ACE. 



Management Task 


Start 


Status 


Description 


Program Management 


August 2001 


Ongoing 


Apply the Customs enterprise life cycle management 
process to the various modernization program activities. 


Enterprise 
Architecture and 
Engineering 


August 2001 


Ongoing 


Updates enterprise architecture and develops Web portal 
and plan for transitioning to the new modernization 
environment. 


Enterprise Process 
Improvement Support 


February 2002 


Ongoing 


Supports integrating all process improvement efforts 
across the enterprise and develops enterprise quality 
assurance program. 


ACE/ITDS Integration 
Support 


February 2002 


Ongoing 


Establishes a strategy for integrating ITDS with ACE, and 
secures stakeholder buy-in. 



Source: GAO. 
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ACE Increment Task 


Start 


Status 


Description 


ACE Requirements and 
Planning — Increment 1 


August 2001 


Completed 
February 2002 


Updates and develops the requirements and desired 
business results and developed the concept of operations 
and project plan for ACE increment 1. 


ACE Increment 1, 
Release 1 
Implementation 


February 2002 


Ongoing 


Designs, develops, and implements ACE increment 1, 
release 1. 


ACE Increment 1, 
Release 2 
Implementation 


August 2002 


Ongoing 


Designs, develops, and implements ACE increment 1, 
release 2. 


ACE Requirements and 
Planning — Increment 2 


February 2002 


Ongoing 


Updates requirements and desired business results and 
develops concept of operations and project plan for ACE 
increment 2. 



Source: GAO. 
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Background 



Customs' Fourth Expenditure Plan Summary 

On November 22, 2002, Customs submitted its fourth ACE expenditure plan, seeking release of $314 million. 



Activities 



Increment 2, release 1 definition, design and development and release 2 definition 

Increment 3 requirements and planning 

Increment 1 operations and maintenance 

eCP ACE program management 

Enterprise architecture and engineering 

CMO costs 

ITDS development and integration into ACE 

National Customs Automation Program (NCAP) and Free and Secure Trade (FAST) 9 
operations and maintenance 

SAP 10 implementation support 

Subtotal 
Management reserve 11 

Total 



Amount $ millions 



$185.9 
$14.8 
$6.1 
$11.9 
$15.7 
$37.5 
$5.4 
$4.5 

$1.5 
$283.4 
$30.6 
$314.0 



* NCAP is the ACE prototype system that has served as a pilot for demonstrating certain ACE functionality, including account-based import processing. FAST is an 
enhancement to NCAP that allows expedited border processing for shipments involving importers, carriers, and commercial drivers who are predetermined to be 
low risk. ACE funding is being used to operate and maintain both NCAP and FAST. 

10 The SAP Implementation Project, a joint project of the Office of Finance and the Office of Information Technology, is intended to provide automated enterprise 
resource planning. The SAP product provides multiple modules, each performing separate, but integrated business functions. Some are used to support Customs' 
financial management, and others may be used to support ACE functions, such as customer relationship management and supply chain management. 

11 Customs has accumulated $34.4 million in management reserve from prior funding releases. As of December 11, 2002, Customs reports that $0.7 million of the 
$34.4 million has been obligated. 
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ACE Increment 2 Release 1 

Customs' fourth expenditure plan includes acquisition activities associated with the second of four ACE increments. 
Specifically, the plan requests $185.9 million for the definition of increment 2, releases 1 and 2, and the design and 
development of increment 2, release 1. Increment 2, release 1, is to be deployed at all ports (truck, rail, sea, and air). 
Increment 2, release 1, is intended to 

• expand account and revenue management capability to high-volume carriers and brokers; 

• enhance query capability and provide a data warehouse; 

• provide consolidated selectivity, targeting, and enforcement functionality; 

• provide access to trade data for four federal agencies; 12 and 

• establish a link to Customs Automated Export System. 13 



12 The four federal agencies are the Food and Drug Administration (Center for Food Safety and Nutrition), Department of Agriculture (Animal and Plant Health 
Inspection Service), Immigration and Naturalization Service (crew release data only), and Census Bureau. 

13 This system allows electronic transmissions of key export data from exporters, exporters' agents, and outbound carriers. 
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Chronology of Prior ACE Expenditure Plans and Funding Releases 

In March 2001, Customs' appropriations committees released to Customs $5 million in stopgap funding for the ACE 
program to sustain CMO operations. 

On March 26, 2001, Customs submitted its first ACE expenditure plan to its appropriations committees, seeking 
release of $45 million to award the modernization contract, sustain CMO operations, and provide the support needed 
to manage the modernization program. Customs' appropriations committees subsequently approved the use of $45 
million, bringing the total ACE funding to $50 million. 

On February 1, 2002, Customs submitted its second ACE expenditure plan to its appropriations committees seeking 
release of $206.9 million to sustain CMO operations; define, design, develop, and deploy increment 1, release 1; and 
identify increment 2 requirements. Customs' appropriations committees subsequently approved the use of $188.6 
million, bringing total ACE funding to $238.6 million. 

On May 24, 2002, Customs submitted its third ACE expenditure plan to its appropriations committees, seeking release 
of $190.2 million to finance the definition, design, development, and implementation of ACE increment 1, release 2. 
Customs' appropriations committees subsequently approved the use of $190.2 million, bringing total ACE funding to 
$428.8 million. 

The following figures chronicle the history of ACE appropriations, expenditure plans, and funding releases. 
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Background 



ACE appropriations, expenditure plans, and funding releases 



FY 2001 Appropriations 
= $130 million 



ipropriations FY21 
} million 

April 2001 
$45 million 

funding . , 

release | March 2002 

' ^ $188.6 million 
funding 

/ ^\ release 



FY2002 Appropriations 
$300 million 






Source: GAO. 
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Background 



Illustrated chronology of ACE expenditure plans, funding releases, and contractual events 



$5M funding 
release 



3/01 



$45M 
funding 
release 



Contract 
awarded 



6/01 



Contractor 
authorized to 
proceed 



Task orders 1 , 
2, 3 started 



9/01 



$188.6M 
funding 
release 



Task order ( 
started 



Task orders 4, 
5, 7 started 



12/01 



3/02 



$190.2M 
funding 
release 



Task order i 
started 



6/02 



9/02 



12/02 



Fourth plan 
submitted 

Source: GAO. 

Cumulative funding through third expenditure plan = $428.8 million. 



First plan 
submitted 



Second plan 
submitted 



Third plan 
submitted 
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Background 



Conceptual View of the ACE Architecture 



Client Tier 



Web& 
Interface Tier 



Enterprise 

Application Applications 
Integration Tier 
Tier 



Data Tier 




Customs & 

Other 
Agencies 



Security 



Source: GAO and Customs. 



The Client Tier includes the user's workstation and external systems that access ACE. 

The Web & Interface Tier provides the interface required to access functionality and data, as well as the 
infrastructure required to support that access. 

The Enterprise Application Integration Tier provides middleware connecting business applications and users across 
Customs. 

The Applications Tier includes the modernized systems (i.e., ACE) and legacy systems (e.g., ACS) that provide the 
automated business applications supporting execution of Customs' business processes. 

The Data Tier provides data storage resources. 
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Legislative Requirement 



Plan meets OMB Capital planning 
and investment control review 
requirements, including OMB 
Circular A-ll, part 3. 



Results of our analysis 



The expenditure plan addresses OMB capital planning and 
investment control review requirements, including OMB 
Circular A-ll, part 3. For example, the expenditure plan 
addresses 

• federal security and privacy regulations; 

• Government Paperwork Elimination Act 
requirements; 

• Government Performance Results Act requirements; 

• how the project will use a performance-based 
management system; and 

• cost, benefit, and schedule information. 



2. Plan complies with Customs' 
enterprise architecture 



■/ The expenditure plan is for, among other things, the 

definition, design, and development of ACE increment 2, 
release 1 (61 percent of requested funding). Customs plans 
to certify that increment 2, release 1, is compliant with 
Customs' enterprise architecture before beginning detailed 
design activities. 
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Le, 


gislative Requirement 




Results of our analysis 


3. 


Plan complies with federal 
acquisition rules, requirements, 
guidelines, and systems acquisition 
management practices. 




The expenditure plan is for, among other things, 
continuation of Customs' acquisition management 
improvement efforts based on the SEI's SA-CMM, as well 
as for implementation of OMB acquisition and investment 
management requirements. 14 


4. 


Plan is reviewed and approved by 
Customs Investment Review Board, 
Treasury, and OMB. 




The Customs and Treasury Investment Review Boards 
were consolidated to form the Joint Capital Investment 
Review Board, which approved the plan on October 15, 
2002. OMB approved the plan on November 12, 2002. 


5. 


Plan is reviewed by GAO. 




Customs delivered its expenditure plan to Congress on 
November 22, 2002. Our review was completed December 
26, 2002. 



" OMB Circular A-130 and A-ll, part 3 (July 2001). 
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Open recommendation: Justify and make investment decisions incrementally, and for each increment, 

• use disciplined investment processes; 

• prepare realistic and supportable benefit expectations; 

• require a favorable return on investment and compliance with Customs architecture before making an 
investment decision; and 

• validate actual costs and benefits once an increment is piloted. 

Customs' actions to address this recommendation are fully consistent and in progress. 



Customs has committed to and defined process controls for justifying and making investment decisions 
incrementally. For ACE increment 2, Customs 

• followed its investment management process, which we determined meets OMB's policy for management 
of federal information resources; 16 

• prepared benefit expectations, which we determined are consistent with OMB's policy for cost benefit 
analyses of federal programs; 16 

• reported a favorable return on investment (benefit/cost ratio for Customs of 1.09 1 ') and plans to certify 
compliance with Customs architecture before beginning detailed design activities; however, this return 
on investment assumes a 4-year implementation schedule, and Customs is planning to extend this 
schedule — this extension could increase costs and delay benefits and thus decrease the benefit/cost ratio; 
and 

• plans to validate actual costs and benefits during post-implementation reviews. 



"OMB Circular A-130. 

10 OMB Circular A-94. 

11 This ratio includes costs and benefits to Customs, exclusive of costs and benefits that are identified as positive to the trade community. 
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Results: Objective 2 



Open recommendation: Before building each ACE release (i.e., beginning detailed design and development), certify 
to Customs' House and Senate appropriations subcommittees that the enterprise architecture has been sufficiently 
extended to provide the requisite enterprise design content and has been updated to ensure consistency and 
integration across business areas. 

Customs' actions to address this recommendation are fully consistent and in progress. 



On July 16, 2002, before beginning detailed design and development of ACE increment 1, release 1, Customs certified 
to its House and Senate appropriations subcommittees that the enterprise architecture had been sufficiently 
extended to provide the requisite enterprise design content and has been updated to ensure consistency and 
integration across business areas. Customs plans to recertify its enterprise architecture before building ACE 
increment 1, release 2 (scheduled for recertification in January 2003), and to continue doing so on all future ACE 
releases. 
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Results: Objective 2 



Open recommendation: Develop and implement a rigorous and analytically verifiable cost estimating program that 
embodies the tenets of effective estimating as defined in SEI's institutional and project-specific estimating models. 

Customs' actions to address this recommendation are fully consistent and in progress. 



On September 26, 2002, Customs hired a contractor to define and implement a cost estimating program for CMO. 
Customs has specified that the program comply with industry best practices and SEI guidance. 18 Customs expects to 
initiate the program in December 2002. As a result, this program was not in place during the development of the cost 
estimates contained in the fourth expenditure plan. 

To validate the reliability of the fourth expenditure plan cost estimate for ACE increment 2, Customs tasked its 
support and modernization contractors with comparing the estimate against SEI criteria for validating software cost 
and schedule estimates. The results of this comparison are shown in the following table. 



SEI Question 


Satisfies? 19 
Fully 1 Partially 


Description 


1. Are the objectives of the 
estimate clear and correct? 

2. Has the task been appropriately J 
sized? 


The objective of the estimate was clearly stated. The tasks and 
activities to be included in the estimate were clearly identified. 
The task was appropriately sized. For example, software sizing 
estimates were determined by requirements, and function point 
counts were performed by certified staff. 20 



18 SEI's institutional and project-specific estimating guidelines are defined in Checklist and Criteria for Evaluating the Cost and Schedule Estimating Capabilities 
of Software Organizations and A Manager's Checklist for Validating Software Cost and Schedule Estimates, respectively. 

19 Fully means that Customs' actions have completely addressed the question; partially means that Customs' actions have not completely addressed the question as 
indicated in the description. 

" The International Function Point Users' Group maintains the recognized industry standard for function point analysis, which is a methodology for software sizing. 
The group offers professional certification for practitioners of function point analysis as Certified Function Point Specialists. 
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SEI Question 



3. Are the estimated cost and 
schedule consistent with 
demonstrated accomplishments 
on other projects? 

4. Have the factors that affect the 
estimate been identified and 
explained? 

5. Have steps been taken to ensure 
the integrity of the estimating 
process? 



6. Is the organization's historical 
evidence capable of supporting 
a reliable estimate? 



7. Has the situation changed since 
the estimate was prepared? 



Satisfies? 

Fully I Partially 



NA 



Description 



The estimated costs are consistent with other projects. For example, 
the cost and schedule models used to develop the estimate were 
calibrated to eCP historical data of other programs. However, 
sufficient project experience has not been acquired to confirm the 
validity and relevance of the selected historical data. 
The factors that affect the estimate were identified and explained. 
For example, uncertainties in software and other parameter values 
were identified, and estimates were developed and documented. 
Steps were taken to ensure the integrity of the estimating process. 
For example, a second software estimating tool, Software Life Cycle 
Model was used for validating the software cost estimate. Cost 
models produced software estimates for normal and compressed 
schedules. However, the cost models do not support schedules with 
the extensive level of parallel development activities that are 
necessary to achieve the proposed ACE schedule. 

Historical evidence was capable of supporting a reliable estimate. 
For example, eCP draws from a database that contains productivity 
data and support cost percentages from a large set of completed 
projects. However, the data did not support the extensive level of 
parallel ACE development activities. 

Because the situation had not changed, this question was not 
applicable to planning estimates. 



We reviewed the contractors' analysis and found no reason to question it. For example, using a different cost 
estimating tool (CostXpert) with eCP's software size estimates and assumptions, we found that the estimate for ACE 
increment 2, release 1, was reasonable. 
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Open recommendation: Immediately develop and implement a CMO human capital management strategy that 
provides both near-term and long-term solutions to CMO's human capital capacity limitations. 

Customs' actions to address this recommendation are partially consistent and in progress. 



We made our recommendation in March 2002, with the intent that Customs specify in its strategy how it would 
accomplish its human capital goals (i.e., who would do what and by when, what resources would be needed, and 
how progress would be measured). 

CMO released a human capital management strategy in October 2002 that specifies high-level commitments, 
including time frames, such as completing vacancy action plans by April 2003, creating an acquisition program 
management career track by April 2004, and defining performance standards by June 2003. 

Based on our analysis of the strategy against 10 human capital management principles drawn from commercial best 
practices, 21 we found that the strategy either fully or partially recognizes each of the principles (see following table 
for a comparison). However, neither the strategy nor supporting documentation identifies, for example, how the 
commitments made in the strategy will be met, including what steps will be taken and what resources are needed to 
execute the steps. Moreover, based on the strategy's timeline, it will be over a year before CMO implements certain 
principles. 

According to Customs officials, development and implementation of its human capital management strategy has 
been delayed because CMO staff have been addressing higher priority areas. 



U.S. General Accounting Office, Human Capital Key Principles from Nine Private Sector Organizations, GAO/GGD-00-28 (Washington, D.C.: January 2000). 
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Human Capital Principles 


Satisfies? 22 


Summary of Comparison 




Fully 


1 Partially 




Treat human capital management as 
being fundamental to strategic 
business management. 






The human capital management strategy ties the organization's 
human capital policies, processes, and practices to the 
modernization mission and strategic goals. 


Integrate human capital functional 
staff into management teams. 






The strategy does not provide for integrating human capital 
functional staff into CMO. However, a human capital staff member 
assisted in developing the strategy. 


Leverage the internal human capital 
function with external expertise. 






The CMO person responsible for human capital management is 
assisted by external expertise from its support contractor. 


Hire, develop, and retain employees 
according to competencies. 






The strategy states that CMO will hire, develop, and retain 
employees according to a planned training program and CMO- 
specific competencies that are yet to be defined. The date for 
completing competency definitions was extended from November 
25 to December 31, 2002. The date for initiating training program 
development was extended from September 30 to December 31, 
2002. 


Hire, develop, and sustain leaders 
according to leadership 
characteristics identified as essential 
to achieving specific missions and 
goals. 






The strategy states that CMO will hire, develop, and sustain 
leaders according to leadership characteristics, which are 
expected to be defined in CMO-specific competencies. 



22 Fully means that the strategy provides for implementing the principle and the actions associated with implementing the principle have been completed. Partially 
means that (1) the strategy states that the principle will be implemented, but does not specify the actions necessary to do so; (2) the strategy describes some, but 
not all, actions necessary to implement the principle, but these actions have been delayed; or (3) the strategy does not state that the principle will be implemented, 
but Customs took some other action related to implementing the principle. 
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Results: Objective 2 


Human Capital Principles 


Satisfies? 

Fully Partially 


Summary of Comparison 


Communicate a shared vision that all 
employees, working as one team, can 
strive to accomplish. 




The strategy states that CMO will develop guidance that is intended 
to communicate a shared vision that all employees, working as one 
team, can strive to accomplish. The strategy states further that this 
guidance will define performance objectives linked to team 
accomplishments. 


Use performance management 
systems, including pay and other 
meaningful incentives, to link 
performance to results. 




The strategy states that CMO will develop and use a performance 
management system, including pay and other meaningful incentives, 
to link performance to results. According to the strategy, 
performance standards will be defined by June 2003 and 
development goals will be set by September 2003. 


Support and reward teams to achieve 
high performance. 




The strategy states that CMO will support and reward teams 
through existing incentives. 


Integrate employee input into the 
design and implementation of human 
capital policies and practices to 
achieve high performance. 




The strategy states that CMO will integrate employee input into the 
design and implementation of human capital policies and practices. 


Measure the effectiveness of human 
capital policies and practices to 
achieve high performance. 




The strategy states that CMO will measure the effectiveness of 
human capital policies and practices. 



Customs' efforts to date are not fully consistent with our recommendation, because (1) they have not been immediate 
and are taking too long to accomplish, and (2) although the strategy commits to doing what we recommended, 
Customs has not developed and implemented key human capital management practices to address the 
recommendation. 

As a result, CMO's ability to manage its human capital resources according to these best practices, and thereby 
ensure that it has the capacity to acquire ACE, is over a year away. 
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Open recommendation: Limit future expenditure plan requests for management reserve funds to 10 percent of the 
total funds requested for the program, or adequately justify any management reserve requests in excess of 10 
percent. 

Customs' actions to address this recommendation are fully consistent and in progress. 



Customs' fourth expenditure plan requests $30.6 million for management reserve, which is 10.8 percent of the total 
funds requested, excluding the management reserve itself. 23 

Management reserves are intended to earmark funding to address project risks, including known risks that, if not 
sufficiently mitigated, become actual problems that require resources to correct. Thus, estimating the amount of 
funding to earmark in the reserve should be closely aligned with the project's risk management program, including 
risk identification and risk probability and impact assessment, and it should be based, in part, on the estimates of 
costs to correct the potential impacts associated with known risks. 

The methodology used to develop the Customs' $30.6 million management reserve request provides for identifying 
ACE risk elements, assessing risk probability and impact, and estimating the cost to correct the potential impacts 
associated with known risks. Moreover, the risk elements are aligned with the risks included in the ACE risk 
management program, meaning that the risk elements used in estimating the reserve are legitimate. As a result, we 
found no reason to question the $30.6 million request. 

The management reserve breakdown by expenditure plan activities is shown in the following table. 



" Customs has accumulated $34.4 million in management reserve from prior funding releases. As of December 11, 2002, Customs reports that $0.7 million of the 
$34.4 million has been obligated. 
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Activity 


Cost 


Reserve 


Percentage 




($ millions) 


($ millions) 


(reserve/cost) 


Increment 2, releases 1 and 2, definitions 


$17.7 


$2.9 


16% 


Increment 2, release 1, design and development 


$168.2 


$23.8 


14% 


Increment 3 requirements and planning 


$14.8 


$0.8 


5% 


Increment 1 operations and maintenance 


$6.1 


$0.2 


3% 


eCP ACE program management 


$11.9 


$0.8 


7% 


Enterprise architecture and engineering 


$15.7 


$1.1 


7% 


CMO costs 


$37.5 


$0.9 


2% 


ITDS development and integration into ACE 


$5.4 


$0.0 


0% 


NCAP operations and maintenance and FAST implementation 


$4.5 


$0.0 


0% 


SAP implementation support 


$1.5 


$0.0 


0% 


Total 


$283.4 


$30.6 


10.8% 
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Open recommendation: Develop and implement process controls for the SEI SA-CMM level 2 key process areas and 
for the level 3 acquisition risk management key process area, and by September 30, 2002, conduct and report to 
Customs' House and Senate appropriation subcommittees on the results of an internal assessment of CMO's 
maturity against these key process areas. 

Customs' actions to address this recommendation are partially consistent and in progress. 



The SEI, recognized for its software engineering expertise, defines key process areas that are necessary for effective 
acquisition management. SEI research shows that not having effective acquisition management processes increases 
the probability of systems not performing as intended and costing more and taking longer than planned. The SA-CMM 
process areas are described in the following table. 



Key process area 


Description 


Software acquisition planning (SAP) 


Ensures that reasonable planning for the software acquisition is conducted and 
that all elements of the project are included 


Solicitation (SOL) 


Ensures that award is made to the contractor most capable of satisfying the 
specified requirements 


Requirements development and 
management (RDM) 


Establishes a common and unambiguous definition of software acquisition 
requirements understood by the acquisition team, system user, and the 
contractor 


Project office management (PM) 


Manages the activities of the project office and supporting contractor(s) to 
ensure a timely, efficient, and effective software acquisition 


Contract tracking and oversight (CTO) 


Ensures that the software activities under contract are being performed in 
accordance with contract requirements, and that products and services will 
satisfy contract requirements 
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Results: Objective 2 


Key process area 


Description 


Evaluation (EVAL) 


Determines that the acquired software products and services satisfy contract 
requirements before acceptance and transition to support 


Transition to support (TTS) 


Provides for the transition of the software products being acquired to the 
software support organization 


Acquisition risk management (ARM) 


Identifies risks as early as possible, adjusts acquisition strategy to mitigate those 
risks, and develops and implements a risk management process as an integral 
part of the acquisition process 



We first recommended that Customs develop and implement the SA-CMM level two key process areas in February 
1999, and in March 2002 we reiterated this recommendation, adding that Customs report to its appropriations 
committees on its progress in doing so by September 2002. 

On September 30, 2002, Customs reported to its House and Senate appropriations subcommittees the results of an 
internal assessment of its efforts to develop and implement SA-CMM key process areas. Customs reported that its 
process development effort was 90 percent complete. 

However, Customs did not report on its implementation of developed processes. Progress in implementing its 
processes as of November 2002 is shown in the following figure and table. 

According to Customs officials, its progress in developing and implementing software acquisition processes is behind 
schedule because resources (i.e., staff) are being diverted to higher priority activities. 

Because Customs has not yet established effective software acquisition processes, it does not have the means to 
adequately ensure that promised ACE capabilities will be delivered on time and within budget. 
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SAP (15) 


SOL (18) 


PM (16) 


RDM (14) 


CTO(17) EVAL(15) 


TTS (16) 


ARM (15) 


Totals 


□ 


Key practices not 
developed and not 
implemented 


1 


1 


2 


1 


3 


2 


12 


1 


18% 


□ 


Key practices 
developed, but not 
implemented 


13 


17 


13 


13 


14 


13 


4 


14 


80% 24 


■ 


Key practices 
developed and 
implemented 


1 





1 

















2% 



Source: GAO. 



100% 



21 This percentage of key practices developed is based on each key practice having equal weight; Customs' method for determining that its process development 
effort was 90 percent complete was based on a method that weighted key practices. 
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Results: Objective 2 



Open recommendation: Address the risks associated with the accelerated ACE acquisition strategy. 
Customs' actions to address this recommendation are fully consistent and in progress. 



Customs' third expenditure plan contained a 4-year accelerated schedule (see figure). Before this, Customs was 
following a 5-year schedule. 



Fiscal Fiscal 
year 2001 year 2002 



Fiscal 
year 2003 



Fiscal 
year 2004 




Fiscal 
year 2005 



Increment 1 release 1 
Increment 1 release 2 
Increment 2 release 1 
Increment 2 release 2 
Increment 3 release 1 
Increment 3 release 2 
Increment 4 



Requirements & planning 
CD Design (preliminary & detailed), develop, deploy 
Source: GAO. 
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Results: Objective 2 



Customs current ACE schedule is not set, although it appears that Customs will not be pursuing the accelerated 4- 
year schedule. Specifically, Customs officials told us that they began rethinking the 4-year schedule because the 
funding needed to support such an accelerated schedule was not received and that they expect to follow a 5-year 
schedule. 2 " According to Customs, a new schedule is expected to be completed and approved by April 2003. At the 
same time, Customs fourth expenditure plan includes a 6-year schedule (see figure). 26 




Increment 1 release 1 
Increment 1 release 2 
Increment 2 release 1 
Increment 2 release 2 
Increment 3 
Increment 4 



LJ Requirements & planning 

! _} Design (preliminary & detailed), develop, deploy 

Source: GAO and Customs. 



25 Actually 5 years, 7 months beginning in March 2002 and ending in September 2007. 

Actually 6 years, 2 months beginning in August 2001 and ending in September 2007. According to Customs, this schedule is intended to portray a high level 
representation of project phases and is not intended to represent a detailed schedule. 
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Observation: Contractor-performed system integration and acceptance testing is conducted in accordance with 
best practices. 



To be effective, test activities should be planned and conducted in a structured and disciplined fashion. Industry best 
practices provide a framework for managing testing activities. This framework includes steps for establishing a 
testing organization and the infrastructure for conducting test activities, and for planning and conducting incremental 
phases of testing, including integration and system acceptance testing. 

The contractor's integration and system acceptance test programs are consistent with best practices. In particular, 
eCP has established a test organization and built an infrastructure for test activities. Additionally, eCP has conducted 
system integration and acceptance tests for ACE increment 1 release 1.1, 21 and these test activities satisfied each of 
the major steps associated with test management best practices, as summarized in the following table. 



Major Test Steps 


Satisfies? 




Integration Test 


System Acceptance Test 


Schedule and plan test 




-/ 


Prepare test procedures 






Define test exit criteria 






Execute tests 






Document test results 






Correct defects 28 






Ensure that test exit criteria are met 







21 ACE increment 1, release 1, consists of two sub-releases, designated releases 1.1 and 1.2. 

28 ACE increment 1, release 1.1, still had system defects at the end of both integration and acceptance testing. However, because the defects were noncritical, and 
defined test exit criteria allow for noncritical defects, this step was satisfied. 
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Observation: Contractor-performed system acceptance test results are positive. 



System Acceptance Testing (SAT) is intended to show that a system performs according to the system specification 
(i.e., performs specified functions within defined performance parameters). Acceptance testing is generally 
conducted by the developing organization (i.e., eCP), with the participation of the acquiring organization (i.e., CMO). 
The criteria for passing SAT generally provide for executing prescribed test cases without the realization of material 
defects. 

All but seven requirements for ACE increment 1, release 1.1, successfully passed their test cases for SAT. The seven 
requirements that did not pass resulted in only noncritical system defects. Accordingly, these defects were deferred 
for correction in future ACE releases. For example: 

• One of the defects related to the requirement for the portal to print search results. This defect was deemed non- 
critical because an acceptable workaround (e.g., selecting the search results and using the Web browser's print 
function) will be used. 

• Another defect related to the requirement that the portal user sessions automatically time out if the session is 
not active after a certain amount of time. This failure resulted in identification of a defect because Customs 
policy requires users to invoke their screensavers or log off when leaving their workstations. However, the 
defect was deemed noncritical because an acceptable workaround will be used (i.e., Customs plans to reinforce 
this policy during the training that will be required before using the system). 

Because all the ACE requirements for increment 1, release 1.1, either passed their tests or, in the case of the seven 
failures, acceptable workarounds were identified, the first ACE release successfully completed SAT. 
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Observation: System defect trends suggest contractor is delivering a quality product. 



One measure of system maturation and quality is downward trends in unresolved system defects, particularly defects 
that are categorized as material. 

Defects are system problems (hardware and software) that require a solution. A defect can be due to a failure to meet 
the system specifications, or it can occur because specified requirements are not complete and correct. To facilitate 
the management of defects identified in testing, eCP documents defects in program trouble reports (PTRs) and 
categorizes them by their severity (impact). The categories are shown in the following table. 



Category 


Description 


Critical (1) 


A problem that makes it impossible to execute a test case 


Severe (2) 


A problem that adversely affects the execution and completion of a test case and for which no 
acceptable workaround can be found 


Moderate (3) 


A problem with an acceptable workaround that allows the test case to be executed 


Minor (4) 


An inconvenience or annoyance that does not affect test case execution 



Data provided by eCP on all unresolved defects for ACE increment 1, release 1.1, show a recent downward trend, 
which suggests that eCP is delivering a quality product (see following figure). To more thoroughly analyze defect 
trends, we requested defect data on critical and severe defects, as the existence of these defect types carry material 
consequences. Customs and eCP could not provide documentation on the status of critical and severe defects in time 
for this briefing because it did not readily exist. Instead, they stated that there were no critical or severe defects as of 
December 16, 2002. 
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Observation: Customs' oversight of contractor testing does not provide for independent verification and validation 
(IV&V). 



Industry test management best practices include steps for effective managerial oversight and control, including 
employing independent verification and validation (IV&V). 

IV&V involves having an independent organization conduct unbiased reviews of test management processes, 
products, and results. The goal is to verify and validate that these meet stated requirements and standards. 
Independence is important in ensuring that the IV&V personnel are unaffected by program management pressure. 

However, Customs' managerial oversight and control of ACE testing does not include IV&V. Customs officials stated 
that they did not establish an IV&V function because they hired a "world class" contractor to develop ACE and have 
required that the contractor be a Software Capability Maturity Model (SW-CMM) 29 level 3 organization. Customs 
officials also stated that they are relying on the contractor's capabilities and reputation as a guarantor of ACE 
quality, 30 and that CMO (Customs and support contractors) oversees the contractor's testing. 
These actions are not an adequate substitute for IV&V, because the oversight being performed by CMO lacks 
independence from program cost and schedule pressures. For large, complex, and inherently risky programs like 
ACE, IV&V is a recognized best practice. Without it, Customs is unnecessarily exposing itself to risks that ACE 
increments will not perform as intended or be delivered on time and within budget. 



" The SW-CMM identifies key process areas that are necessary to effectively develop software-intensive systems. Level 3 is the third level of the SW-CMM's five- 
level scale; achieving this level means that an organization has defined and implemented effective software engineering and management processes across all 
projects. 

30 In its comments on our report, Customs modified this statement. Customs' modified statement is reflected in the letter transmitting this briefing. 

40 



Page 46 



GAO-03-406 Customs Service Modernization 



Appendix I 

Customs' Fourth Automated Commercial 
Environment (ACE) Expenditure Plan 



GAP 

Accountability * Integrity ' Reliability 



Results: Objective 3 



Observation: Potential for ACE infrastructure to support other Department of Homeland Security applications 
could affect ACE cost and schedule. 



Customs' current cost and schedule commitments for its ACE program, as previously described in this briefing, are 
based on a set of Customs' enterprise infrastructure and ACE-specific requirements. As with any such program, 
changes to scope will affect these commitments. Also, a meaningful understanding of the magnitude of these cost 
and schedule effects requires a complete understanding of the scope changes. 

Customs is one of several agencies to be merged into the Department of Homeland Security and is to support the 
new department's border and transportation security mission area. Customs has proposed to the Office of 
Homeland Security that the ACE infrastructure be used by the various agencies involved in carrying out this 
mission. According to Customs, its infrastructure is being designed and developed in a way to make it expandable 
and extendable, and thus able to support other border and transportation security applications. However, Customs' 
Chief Information Officer told us that Customs has yet to receive an official response from the Office of Homeland 
Security on this proposal, and a meaningful understanding of the effect of the new department's use of the ACE 
infrastructure is not yet known, because the detailed engineering analysis needed to understand the associated 
program scope changes (e.g., workload sizing, security) has not been done. 

As a result, while the potential exists for the cost and schedule commitments defined in the ACE expenditure plan 
to change, the magnitude of such changes is not yet known. 
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Customs' fourth expenditure plan satisfies its legislative conditions and is fully consistent with most of our open 
recommendations. Nevertheless, Customs has moved slowly in addressing our recommendation for immediately 
correcting its CMO human capital weaknesses, thus far only having made statements calling for implementing human 
capital management best practices. Compounding this, Customs has not satisfied the intent of our recommendation 
for disclosing to its appropriations committees where it stands in putting in place acquisition management best 
practices. Moreover, its progress in implementing these practices has been slow, and where it stands today continues 
to put ACE at risk. 

Both of these areas — people and processes — are fundamental components of an organization's capability to acquire a 
system like ACE. Despite this, Customs has not given either sufficient priority, instead focusing to date on expediting 
eCP development activities and expressly compensating for its lagging acquisition capabilities by relying on eCP, 
without the benefit of IV&V, to be the guarantor of ACE quality. 

Customs' approach of offsetting weaknesses in its acquisition process and its people management by relying on its 
contractor has to date not produced visible and measurable cost, schedule, and quality shortfalls, as evidenced by 
reported defect trends and system acceptance test results for ACE increment 1, release 1. However, research has 
linked weaknesses in an organization's ability to acquire software-intensive systems with such shortfalls. Thus, until 
Customs fully addresses its weaknesses in its acquisition process and its people management, it will be at risk of 
experiencing such shortfalls. This risk is increased by the potential changes in store for ACE should its expanded role 
in the Department of Homeland Security be realized. Therefore, it is critical that Customs, as the acquirer of ACE, 
treat advancing its acquisition capabilities as a priority matter. 
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To ensure that Customs has the requisite capability to manage its ACE acquisition, we recommend that the Customs 
Service Commissioner designate strengthening CMO human capital and acquisition processes as priority matters. To 
this end, we recommend that the Commisssioner direct the Chief Information Officer (CIO) to immediately 

• develop and implement the missing human capital management practices discussed in this briefing, and until 
this is accomplished, report to its appropriations committees quarterly on the progress of its efforts to do so; 

• develop and implement each of the missing SEI SA-CMM practices for the key process areas discussed in this 
briefing, and until this is accomplished, report to its appropriations committees quarterly on the progress of its 
efforts to do so; and 

• establish an IV&V function to assist Customs in overseeing eCP efforts, such as testing. 

Additionally, we recommend that the Commissioner take steps, as appropriate in light of Customs' merger into the 
Department of Homeland Security, to have future ACE expenditure plans specifically address any proposals or plans, 
whether tentative or approved, for extending and using ACE infrastructure to support other homeland security 
applications, including any impact on ACE of such proposals and plans. 
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We provided Customs a draft of this briefing and discussed its contents with Customs' Assistant Commissioner for 
Information and Technology (CIO) and with CMO and eCP officials. In its oral comments, Customs agreed with our 
findings, conclusions, and all but one recommendation. Regarding our recommendation to employ IV&V, Customs 
agreed that it does not currently have an IV&V function or agent. Customs also agreed to take the recommendation 
under advisement. However, Customs stated that to date it has satisfied the intent of IV&V through various process 
management controls practiced by eCP, such as risk management, metrics collection, and test management, and 
through CMO oversight of eCP. We do not dispute the performance of these activities by eCP and CMO. However, 
none satisfies the express purpose of IV&V, which is to provide a perspective on project performance independent of 
eCP and CMO. In light of CMO's current acquisition management capabilities, both people and processes, we strongly 
believe that because of the magnitude and complexity of ACE, appropriately targeted IV&V activities are a necessary 
measure to reduce acquisition management risk. 
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U.S. Customs Service 

Memorandum 



DATE: February 13, 2003 



MEMORANDUM FOR RANDOLPH C. HUE 

U.S. GENERAL ACCOUNTING OFFICE 

FROM: Acting Director 

Office of Planning 

SUBJECT: General Accounting Office (GAO) Draft 

Report: Customs Service Modernization 
Automated Commercial Environment (ACE) 
Progressing, but Further Acquisition Management 
Improvements Needed 



Thank you for providing us with a copy of your draft report and the 
opportunity to discuss the issues in this report. 

Overall, we are pleased with the GAO report. GAO concludes that 
Customs has met the legislative requirements imposed by Congress and 
continues to progress in addressing prior GAO recommendations. GAO 
also reports a positive assessment of the testing practices and results of 
the ACE contractor. 

GAO characterizes progress in implementing a human capital strategy 
and key acquisition management practices as slow and recommends 
quarterly reporting of progress to the appropriations committees. While 
we certainly agree with the recommendations, we briefly note actions 
taken that show that Customs continues to work diligently to address 
these two critical program management dimensions. 

Customs continues to implement its Modernization Strategic Human 
Capital Management Plan (HCMP) in accordance with the GAO 
principles. Priority attention was given to gaining Commissioner approval 
for an expanded Customs Modernization Office (CMO) organizational 
tradition structure and filling these positions. This has been accomplished, and a 
talented cadre of executives is trained and functioning to oversee the 
★ Modernization task orders. In addition, our quarterly report will show that 

we have taken responsive actions across the range of GAO principles, 
service such as the following: 
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• Integration of human capital functional staff into management 
teams through the inclusion of the OIT Resources Management 
Group (RMG) in the HCMP implementation and through 
participation of the Director of the RMG and Customs Office of 
Human Resources Management in the Modernization 
Management Team. 

• Issuance of the new "Customs Leadership Development Guide" in 
December 2002 that clearly identifies a comprehensive leadership 
competency model, as well as a Customs Succession 
Management System. 

• Establishment of performance plans for the CMO staff in October 
2002, and the inclusion of Modernization objectives in the 
performance plans of an array of OIT executives and managers. 

• CMO recognition of staff for their contributions at a quarterly 
ceremony with both cash and non-traditional awards. 

The CMO is also continuing to implement key acquisition practices. Its 
January assessment found substantial progress in implementing key 
practices, but Customs agrees that much more progress is needed. To 
ensure that progress is made: 

• CMO and other Customs managers and personnel have been 
assigned responsibilities to manage and implement the acquisition 
processes and have been trained on the process improvement 
goals, timeframes, and their roles in achieving these goals. 

• The CMO continues to internally assess progress against the CMO 
process improvement goals. Additionally, Customs is contracting 
with the Software Engineering Institute (SEI) to conduct both rated 
and non-rated Software Acquisition - Capability Maturity Model 
(SA-CMM) assessments to measure progress towards becoming a 
Level 2 organization. 

• Deficiencies identified in the process asset audit have been 
distilled into Corrective Action Requests (CARs) that are tracked in 
an OIT database. The CMO is aggressively working to close these 
CARs and continues to move as rapidly as possible through 
process implementation. 

These are examples of the progress Customs will include in its quarterly 
report. 

OIT also agrees to implement actions responsive to GAO's 
recommendation relative to an independent verification and validation 
function (IV&V). However, OIT wishes to modify the characterization of 
its IV&V function provided to GAO in the short timeframe we had for a 
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response. To state that we rely upon the contractor's reputation and 
maturity level as guarantors of system quality is not reflective of the range 
of practices the CMO has implemented as part of its integrated approach 
to IV&V. Such practices have included: 

• Independent review of the e-Customs Partnership (eCP) 
requirements products by end users. 

• Involvement of the OIT Software Development and Infrastructure 
Services Divisions in conducting IV&V of the technical design and 
implementation products. 

• Engagement of the MITRE Corporation, as the CMO's Federally 
Funded Research and Development Center (FFRDC), to provide 
technically knowledgeable and independent reviews of all critical 
eCP products. 

These practices represent elements of an Integrated Independent 
Verification and Validation (l 2 V 2 ) function that enables us to accomplish 
the needs of a complex Modernization program. Customs recognizes 
that these functions could benefit from being formalized into a process. 
As a result, where appropriate, Customs will strengthen and formalize this 
process, as well as establish formal accountability for the l 2 V 2 function, 
including formal sign-off. MITRE, as the Customs FFRDC, will take the 
lead in reviewing, documenting, and strengthening the l 2 V 2 process. 

If you have any questions regarding these comments, please contact 
Ms. Michele Donahue at (202) 927-0957. 




Brenda B. Smith 



Attachment 
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U.S. Customs Service 
General Accounting Office (GAO) Review of Fourth Automated Commercial 
Environment (ACE) Expenditure Plan 

Recommendation 1 : Develop and implement the missing human capital 
management practices discussed in this briefing, and until this is accomplished, 
report to its (Customs) appropriations committees quarterly on the progress of its 
(Customs) efforts to do so. 

Response: 

Customs concurs with GAO's recommendation to implement the actions 
identified in the Modernization Strategic Human Capital Management Plan 
(HCMP) and further develop the Customs Modernization Office (CMO) human 
capital practices. Customs also shares GAO's desire to speed implementation 
and thereby mitigate program risks. Customs will report quarterly on its progress 
in implementing its HCMP. 

Milestone Date: March 31 , 2003 for CMO concurrence on Job and 

Core Competencies 



Recommendation 2: Develop and implement each of the missing Software 
Engineering Institute (SEI) Software Acquisition— Capability Maturity Model (SA- 
CMM) practices for the key process areas discussed in this briefing, and until this 
is accomplished, report to its (Customs) appropriations subcommittees quarterly 
on the progress of its efforts to do so. 

Response: 

Customs agrees with GAO on the immediate need to develop and implement all 
of the SEI SA-CMM practices in the key process areas, and believes substantial 
progress is being made toward that end. Customs will provide quarterly progress 
reports to Customs appropriations committees in accordance with Congressional 
direction. 

Milestone Date: April 2003 for updated process asset audit 
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Recommendation 3: Establish an independent verification and validation 
(IV&V) function to assist Customs in overseeing contractor efforts, such as 
testing. 

Response: 

The opportunity afforded by the comment period on the GAO draft report leads 
the Office of Information and Technology (OIT) to modify the position it took in 
responding to GAO's short turnaround request of December 26, 2002 for a 
characterization of its IV&V activities. OIT believes it has in place a number of 
elements of an Integrated Independent Verification and Validation (l 2 V 2 ) function 
that conforms to contemporary practices and has provided value in terms of 
reduced risks. However, Customs recognizes that these functions could benefit 
from being formalized into a process. As a result, where appropriate, Customs 
will strengthen and formalize this process, as well as establish formal 
accountability for the l 2 V 2 function, including formal sign-off. MITRE, as the 
Customs Federally Funded Research and Development Center (FFRDC), will 
take the lead in reviewing, documenting, and strengthening the I V 2 process. 

Milestone Date: May 30, 2003 for an l 2 V 2 strategic plan 

July 31 , 2003 for documented l ? V 2 process 



Recommendation 4: Take steps, as appropriate in light of Customs merger 
into the Department of Homeland Security, to have future ACE expenditure plans 
specifically address any proposals or plans, whether tentative or approved, for 
extending and using ACE infrastructure to support other homeland security 
applications, including any impact on ACE of such proposals and plans. 

Response: 

Customs recognizes the importance of apprising the appropriations committees 
of any changes in ACE plans that might arise from merger into the Department of 
Homeland Security. However, Customs questions the appropriateness of the 
GAO recommendation that "tentative proposals or plans" be included. Customs 
will continue its practice of providing the appropriations committees with 
expenditure plans that are well justified and are vetted and approved across the 
Executive Branch. When the ACE infrastructure is used or extended to support 
other Homeland Security applications, this will be noted in the appropriate 
Expenditure Plans. 

Milestone Date: August 2003 for release of Expenditure Plan to the 

Congress and GAO 



(310251) 
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